Skip to content

Fix '#' inside string literals being lexed as a comment - #22

Merged
FallenAngel97 merged 1 commit into
FallenAngel97:masterfrom
lovesegfault:fix-hash-in-string
Jul 5, 2026
Merged

Fix '#' inside string literals being lexed as a comment#22
FallenAngel97 merged 1 commit into
FallenAngel97:masterfrom
lovesegfault:fix-hash-in-string

Conversation

@lovesegfault

Copy link
Copy Markdown
Contributor

Fixes #21.

Problem

The comment token runs to end of line, so when the lexer sits at the interior of a string like "#fff", the comment token (5+ chars, through the closing quote and beyond) outmatches the string-content token (4 chars, stops at the quote) under the longest-match rule. The string interior is consumed as a comment and the parse of everything after it is corrupted — with a later string in the file, the quotes re-pair across rules:

package p

color := "#fff"

allow := true

parses today with #fff" as a (comment) inside an ERROR.

Fix

Give the quoted_string and raw_string content tokens lexical precedence (prec(1)) over the comment token — exactly the precedence the two interpolated string forms (interpolated_string_double, interpolated_string_raw) already carry, which is why $"#..." was unaffected. Lexical precedence takes priority over match length, so string content wins inside string interiors, while comments after strings on the same line still lex normally (string-content tokens aren't valid outside string interiors):

x := "#a" # real comment

keeps (quoted_string) followed by (comment).

src/ is regenerated with tree-sitter generate.

Tests

New test/corpus/strings.txt covering # in quoted strings, # in raw strings, and a real comment following a string that contains #. Full corpus: 42/42 passing.

@FallenAngel97

Copy link
Copy Markdown
Owner

Hello @lovesegfault !

May I ask you to update this PR as well?
Thanks!

The comment token runs to end of line, so at a position like the
content of "#fff" it outmatched the string-content token on length
and the string interior was consumed as a comment, corrupting the
parse of everything after it (with a later string in the file the
quotes re-pair across rules).

Give the quoted_string and raw_string content tokens lexical
precedence over the comment token, mirroring the precedence the two
interpolated string forms already had. Comments after strings on the
same line still lex normally, since the string-content tokens are not
valid outside string interiors.

Adds corpus tests for '#' in quoted and raw strings and for a real
comment following a string that contains '#'.

Fixes FallenAngel97#21
@lovesegfault

Copy link
Copy Markdown
Contributor Author

done!

@FallenAngel97

Copy link
Copy Markdown
Owner

Thank you, @lovesegfault , I see that tests have passed, therefore all good.
I will merge it!
Great work!

@FallenAngel97
FallenAngel97 merged commit c862d32 into FallenAngel97:master Jul 5, 2026
1 check failed
@lovesegfault
lovesegfault deleted the fix-hash-in-string branch July 5, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

'#' inside a string literal is lexed as a comment start

2 participants